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Abstract 


This paper describes the Controller Acceptability Study 1 (CAS-1) experiment that was conducted by 
NASA Langley Research Center personnel from January through March 2014 and presents partial CAS-1 
results. CAS-1 employed 14 air traffic controller volunteers as research subjects to assess the viability of 
simulated future unmanned aircraft systems (UAS) operating alongside manned aircraft in moderate- 
density, moderate-complexity Class E airspace. These simulated UAS were equipped with a prototype 
pilot-in-the-loop (PITL) Detect and Avoid (DAA) system, specifically the Self-Separation (SS) function 
of such a system based on Stratway+ software to replace the see-and-avoid capabilities of manned aircraft 
pilots. A quantitative CAS-1 objective was to determine horizontal miss distance (HMD) values for SS 
encounters that were most acceptable to air traffic controllers, specifically HMD values that were 
assessed as neither unsafely small nor disruptively large. HMD values between 0.5 and 3.0 nautical miles 
(nmi) were assessed for a wide array of encounter geometries between UAS and manned aircraft. The 
paper includes brief introductory material about DAA systems and their SS functions, followed by 
descriptions of the CAS-1 simulation environment, prototype PITL SS capability, and experiment design, 
and concludes with presentation and discussion of partial C AS-1 data and results. 

CAS-1 results included a total of 1 176 controller assessments of all encounter geometries and HMD, and 
showed that a SS HMD value of 1.5 nmi had the highest acceptability rating with a striking degree of 
agreement among the controller volunteers. More generally, most controllers assessed HMD values 
between 1 and 2 nmi to be acceptable across all encounter geometries. There was more variability in 
controller assessments of 0.5 nmi HMD encounters but a significant number of these assessments 
considered this HMD value to be too small and potentially unsafe. HMD values larger than 2 nmi were 
generally assessed as increasingly disruptive to orderly traffic flow. These results should be useful to 
inform the development of operational performance standards for DAA SS functions. For example, it 
may be appropriate for standards to specify that SS functions should always indicate that SS maneuvering 
is necessary for encounters that will result in projected HMD values less than 1 nmi, and never indicate a 
required SS maneuver if the projected HMD is greater than 2 nmi. 

All 14 air traffic controller volunteers were favorably impressed with the PITL SS concept as simulated 
and presented to them, and considered the concept to be viable from an ATC perspective under the 
assumption that acceptable SS HMD values are employed. CAS-1 staff pilots also assessed the concept 
favorably and considered the simulation time valuable in identifying enhancements for future refinements 
of the concept. These controller and pilot assessments were significant since C AS-1 was the first 
sustained opportunity to exercise the Stratway+ instantiation of the PITL SS concept in a realistic 
controller-pilot simulation that was representative of the moderate-complexity, lower-altitude Class E 
airspace that will pose one of the most challenging environments for DAA-equipped UAS. 

Introduction 

As described in references [1], [2], [3] and other documents, a Detect and Avoid (DAA, also known as 
Sense and Avoid or SAA) system is envisioned to have up to two functions: a self-separation (SS) 
function to enable its associated Unmanned Aircraft System (UAS) to stay well clear of other aircraft, and 
a possibly-optional collision avoidance (CA) function to prevent collisions if all other means of separation 
fail. The SS and CA functions are further envisioned to each be comprised of the following sub- 
functions: Detect, Track, Evaluate, Prioritize, Declare, Determine, Command and Execute (references [1- 
4]; see Figure 1). Some of these sub-functions may be performed by automation and some by the UAS 
pilot, depending on the function (SS or CA) and the overall DAA architecture. As described in references 
[2], [4] and subsequently [5], the SS function is envisioned to allocate at least the Determine and 
Command sub-functions to the UAS pilot, consistent with current manned- aircraft see-and-avoid 
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operations where the pilot in command (PIC) is responsible for determining and commanding appropriate 
avoidance action, if any, based on relevant visual information and ATC communication. The SS sub- 
functions preceding Determine will collectively provide the UAS pilot with information elements and/or 
decision aids sufficient to replace the missing out-the-window visual information and enable the pilot to 
make safe and efficient SS maneuver decisions. 


Target/Destination 


Self-Separation 

Threshold 


Collision 

Point 


Intrude^ 



Execute 
Command 
Determine 
Declare 
Prioritize 
Evaluate 
Track 
Detect 


Figure 1. DAA Sub-Functions, Thresholds and Volumes 


As shown in Figure 1, the SS and CA functions will each have a defined space/time threshold - a self- 
separation threshold (SST) and collision avoidance threshold (CAT), respectively - which will cause the 
function to Declare that avoidance maneuvering may be necessary if an intruder aircraft crosses the 
respective threshold (note that for convenience the DAA community typically references these thresholds 
or boundaries in the singular, i.e., “the SST” or “the CAT” although in practice a uniquely-shaped SST 
and CAT boundary would be computed for each unique intruder encounter geometry, based on the time 
and space parameters defining the respective thresholds). For a representative CA function such as TCAS 
II (references [6] and [7]) an intruder crossing the CAT would result in presentation of visual and aural 
Resolution Advisory (RA) warning declarations to the flight crew (e.g., “Climb, Climb”) with the 
objective of avoiding collision with the intruder, or more precisely, to avoid entry of the intruder into the 
collision volume (CV) around the Ownship. This CV is typically defined as a cylinder centered about the 
Ownship with a radius of 500 feet and height of 200 feet, and the CV penetration rate is considered a 
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performance measure for the effectiveness of various TCAS II versions and other CA functions. 

Reference [2] proposed the definition of a self-separation volume (SSV) to be used similarly for the SS 
function; that is, the SS function would provide guidance and/or alerts sufficient to enable the PIC to keep 
intruders clear of the SSV, and the SSV penetration rate would be considered a performance measure for 
the SS function (reference [3] subsequently proposed a similar concept but labeled the SSV functional 
boundary as Well Clear Violation (WCV)). 

Reference [2] asserted that the SSV should be large enough to avoid (or minimize): 1) corrective RA 
issuance by TCAS-equipped intruders (or by a TCAS CA system onboard the Ownship); 2) safety 
concerns for air traffic control (ATC) personnel; and 3) undue concern for pilots of proximate see-and- 
avoid aircraft. The SSV should also be small enough to minimize disruptions to traffic flow. 
Determination of minimum and maximum acceptable SSV sizes will inform the design space for required 
DAA surveillance accuracy and for acceptable overall SS function performance. 

In order to minimize TCAS corrective RA issuance, reference [2] proposed a SSV functional shape and 
minimum size based on the TCAS II corrective RA CAT, with the shape and size parameterized by 
threshold values of the TC AS-like variables of modified tau (ModTau), distance modification (DMOD), 
projected horizontal miss distance (HMD) at closest point of approach (CPA), vertical threshold (ZTHR) 
and time to co-altitude (TCOA). This parameterized SSV shape has been codified in software along with 
a predictive algorithm (Stratway+) that can provide SS maneuver guidance to the UAS PIC to keep this 
SSV clear of intruders (references [8] though [11]), but several challenges still remain. Two of the 
challenges were addressed by the Controller Acceptability Study 1 (CAS-1) experiment, conducted in 
Winter-Spring 2014 by NASA Langley Research Center personnel and described in this paper: 

1. The pilot-in-the-loop (PITL) SS concept described inreferences [2], [4] and [5] had not been tested 
in a human-in-the-loop ( HITL) simulation representative of challenging cur rent-day National 
Airspace System (NAS) operations with actual pilots and air traffic controllers; and 

2. Threshold values of SSV para meters such as HM D that were generally acceptable to ATC 
personnel, both for safety perceptions and for orderly and efficient traffic flow in a realistic ATC 
environment, were u nk nown. 

The following sections of this paper describe the C AS-1 simulation environment, PITL SS concept 
instantiation in the simulation, and experiment as designed to address these two challenges, followed by 
additional sections to describe some of the CAS-1 results and conclusions. 


Simulation Environment and PITL SS Concept Instantiation 


The CAS-1 simulation environment was based on the Multi Aircraft Control System (MACS) software 
platform. MACS was originally developed by Dr. Tom Prevot and his team at NASA Ames Research 
Center (references [12] and [13]), and this very capable software platform enables HITL simulations of 
realistic air traffic scenarios with many aircraft operating in structured airspace representative of today’s 
NAS. A MACS simulation environment is typically comprised of multiple PC-based workstations 
networked together with each station running an instantiation of MACS and staffed by an operator. The 
operators can be simulation staff, pilots, and/or controllers, and the MACS instantiations can be 
configured to present simulation manager/traffic generator station, aircraft station, or ATC station 
simulations to the respective operators. The simulation manager/traffic generator station initiates all 
aircraft into the simulation via a pre-defined scenario script, enables the transfer of some of these aircraft 
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to pilots at aircraft stations via script or human operator intervention (other aircraft are auto-flown via pre- 
defined flight plans), enables manual or automated removal of aircraft from the scenario (e.g., upon 
landing) and allows for overall scenario control by the operator. The aircraft stations can be set up either 
to present displays/controls of a single aircraft (such as for a P1TL study where the pilot is the subject of 
the experiment) or to enable control/monitoring of multiple aircraft. The multi- aircraft station capability 
is used to enable efficient controller-in-the-loop simulations, so that several staff pilots, each at a multi- 
aircraft station, can collectively simulate many participating aircraft in a subject controller’s sector. The 
ATC stations can simulate either a terminal radar approach control (TRACON) station (i.e., STARS, or 
Standard Terminal Automation Replacement System) or an air route traffic control center (ARTCC) en 
route station (i.e., DSR, or Display System Replacement). Multiple ATC sectors can be simulated, so that 
in a controller-in-the-loop study such as CAS-1 the ATC subject would control traffic at an ATC station 
simulating his or her subject sector, and other staff controllers would operate other ATC stations 
simulating the adjacent TRACON and/or ARTCC sectors. 

The MACS version used for CAS-1 was enhanced with additional capabilities developed by NASA 
Langley personnel to enable simulation of DAA/SS-equipped UAS and the PITL SS concept. These 
capabilities included the addition of UAS vehicle models; configurable simulation of command-and- 
control (C2) link delays; Automatic Dependent Surveillance - Broadcast (ADS-B) In/Out surveillance 
modeling at the message level; and prototype SS guidance capability with associated UAS Ground 
Control Station (GCS) pilot displays, based on Stratway+ software (references [8-11]) and modifications 
to the MACS multi-aircraft station capability, respectively. An extensive multi-channel voice 
communication system was also developed, to simulate aircraft-controller party-line communication on 
the subject sector frequency with configurable delay for UAS aircraft, and also to simulate the land- line 
communication channels between adjacent controllers. These additional capabilities, as they were 
configured for CAS- 1 , are briefly described next. 

A number of UAS vehicle dynamic models were developed by Intelligent Automation, Inc. (LAI) in 2012- 
2013 under contract with NASA Langley and incorporated into the MACS version used for CAS-1, and 
three of these models were used in the CAS-1 simulations. These three models approximated the typical 
operational (versus aerodynamic-performance-limited) performance rates and speeds of the General 
Atomics Predator A and Predator B/Reaper and the Northrop-Grumman Global Hawk aircraft when using 
a terrestrial (versus satellite) C2 link. The simulated C2 link delay was set to zero for CAS- 1 , which also 
more closely simulates the performance of a terrestrial versus satellite C2 link. The voice communication 
delay for CAS-1 was also configured to zero for UAS and all other simulated aircraft, although the actual 
delay was approximately 100-150 milliseconds due to the voice-over-intemet-protocol (VoIP) 
implementation of the communication simulation. This is again consistent with typical nominal 
performance of terrestrial C2 links that also carry relayed voice communications. 

The ADS-B In/Out surveillance modeling capability was used in CAS-1 to simulate the Detect and Track 
sub-functions of the UAS SS function. The modeling capability itself is extensive and includes modeling 
at the message level (versus assembled report level) of both 1090 MHz Extended Squitter (1090-ES)- and 
Universal Access Transceiver (UAT)-equipped ADS-B Out aircraft and associated ADS-Rebroadcast 
(ADS-R) spatial-limit and delay modeling, as well as modeling of Traffic Information Service-Broadcast 
(TIS-B) capability (for detection of unequipped intruders) and various levels of 1090 MHz interference 
modeling. However, for CAS-1 the model was configured such that all aircraft were 1090-ES ADS-B 
In/Out-equipped with ideal detection accuracy, no 1090 MHz interference, and unlimited detection range. 
That is, ideal SS Detect and Track sub-function performance was assumed and simulated for CAS-1. 

SS guidance capability was provided to the UAS pilot in CAS-1 by Stratway+ software. The functional 
parameterized shape of the SSV (references [2, 8-11]) was codified within Stratway+ as well as default 
threshold values for the parameters ModTau, DMOD, HMD, ZTHR and TCOA. These default threshold 
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values were identical to those for TCAS II (reference [7]) but could be overridden via explicit inputs to 
Stratway+, and in fact for CAS- 1 the threshold values for DMOD and HMD were manipulated as 
independent variables (IVs) and explicitly input to Stratway+ for all SS encounters. Additional inputs to 
Stratway+ included Ownship position, velocity, winds, and performance rate data (e.g., nominal turn rate, 
from the vehicle model); intruder positions and velocities; and a specified SS look-ahead time (LAT) 
value. Stratway+ used these data to compute conflict prevention bands (“bands”) for heading, airspeed 
and vertical speed. These computed bands, if any, would then be displayed on the pilot’s heading, 
airspeed, and vertical speed indicators, respectively, to show which respective values would cause one or 
more intruders to penetrate the SSV within the specified LAT. LAT was set to 70 seconds for CAS-1. 


Figure 2 shows a photograph of the navigation display (ND) of a MACS multi-aircraft UAS GCS station 
during a simulation with multiple aircraft near the (currently-selected) Ownship, depicted as the bold 
white chevron at the center of the range rings. The view is track-up but for CAS-1 the wind field was 
always assumed zero, so track and heading are the same (165 degrees magnetic in Figure 2) as are 
groundspeed (GS) and true airspeed (TAS) (157 knots in this case). The range is set to 10 nautical miles 
(nmi), indicated by the white “5” next to the half-scale range ring, and the closest intruder laterally is 
shown at the 9 o’clock position and 3 nmi range, westbound but 2800’ above the Ownship and thus not a 
factor (note that MACS colors intruders white if within 500’ vertically of Ownship, and blue or green if 
more than 500’ above or below, respectively). Stratway+ amber heading bands are visible from 174 to 
273 degrees and are caused by the co-altitude eastbound intruder (N5457B) at 2 o’clock and 5 nmi. 



Figure 2. SS Heading Bands Example: Crossing intruder, no conflict (yet) 


5 



The Stratway+ heading bands in Figure 2 can be interpreted by the UAS pilot as follows: the current 
heading of 165 degrees is conflict-free from a SS perspective, but the new heading after the planned turn 
at the VELCI waypoint will cause the SSV to (just) be penetrated by an intruder (N5457B in this case) 
within the LAT of 70 seconds. A slight modification to the right turn, for example to only 173 degrees, 
should avoid SSV penetration, but further right turns will cause SSV penetrations, up until a very 
significant right turn to 273 degrees or greater. Note that this very significant right turn, while probably 
not operationally desirable, can still be made at the nominal turn rate from the vehicle performance model 
(approximately 2 degrees per second in this case) and the SSV will not be penetrated, if the turn is 
Commanded within the (configurable) reaction time allocation for the pilot (this reaction time is also an 
input to Stratway+ for use in bands computations and was set to 5 seconds for CAS-1). 



Figure 3. SS Heading Bands Example: Crossing intruder, with conflict after turn at VELCI 

Figure 3 shows a photograph of the ND a short time after the situation in Figure 2 and zoomed to the 5 
nmi scale, after the pilot ignored the heading bands and executed the unmodified turn at VELCI. 
Stratway+ has now “Declared” a SS conflict with N5457B and its chevron is now colored amber to so 
indicate that it will penetrate the SSV within the LAT. A slight left turn to 173 degrees or less will still 
result in passing ahead of the intruder and avoiding the SSV penetration, but as the encounter has 
progressed the option to turn right and go behind is no longer available. The vertical speed band is 
presented on the vertical speed tape of the Primary Flight Display (PFD) and is not shown here, but in this 
example briefly indicated that a high-vertical-rate climb or descent would have also avoided SSV 
penetration, by achieving ZTHR of vertical separation. Stratway+ showed this climb or descent option 
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becoming unviable via the vertical speed band shortly after the situation shown in Figure 3, as the 
encounter progressed further and the required vertical rate would exceed the vehicle’s performance 
capability. Note that in this example a slight left-turn modification to the flight plan after VELCI is the 
obvious best operational choice for avoiding SSV penetration by the intruder (i.e., “remaining well 
clear”), but a key aspect of the PITL SS concept described here is to present the UAS pilot with all 
available options - left, right, up, down, even a possible speed change - and allocate the SS Determine 
and Command sub-functions (i.e., what action to take, if any) to the pilot and his or her judgment in 
concert with ATC coordination. 

In addition to the Stratway+ bands, UAS pilots in CAS-1 were presented with an alphanumeric display 
which showed a list of all intruders within 15 nmi of the Ownship and on a converging course. An 
example of this display is shown in Figure 4. The top of the alphanumeric display shows the current 
threshold values of ZTHR and HMD if they have been overridden (-999999 otherwise) so in this case 
Stratway+ is presenting bands for a SSV with 2.5 nmi HMD (note that for CAS-1 and all prior and 
subsequent work to date, the HMD and DMOD thresholds were always set equal to each other, so in this 
example the DMOD threshold is also 2.5 nmi). The list shows converging intruders by call sign along 
with their current range, estimated range at CPA and estimated time to CPA. These last two values 
proved useful to UAS pilots in achieving encounters with very precise HMD, and gave them an 
approximate idea of how much time they had left to negotiate a SS maneuver with ATC before SSV 
penetration was inevitable. 



Figure 4. CAS-1 UAS GCS Display of Nearby Converging Intruders 
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Both UAS and manned aircraft pilots in CAS-1 were also provided with a separate moving map display 
for enhanced situation awareness. An example of this display is shown in Figure 5 with a close-up view 
shown in Figure 6. This example shows the centered Ownship position overlaid on a North-up view of a 
Visual Flight Rules (VFR) Sectional Chart representing the current airspace, but additional options 
included a Track-up view and various satellite imagery or street map underlays. The airspeed tape, 
altitude tape and vertical velocity indicator (VVI) were positioned on the left and right sides of the 
moving map, respectively, as can be seen in Figure 5, and a heading indicator rose was drawn around the 
Ownship position as can be seen more clearly in Figure 6. Intruder aircraft were also depicted on the 
moving map, such as the intruder shown near the upper right comer of Figure 6 over Rockwall Aiiport, 
and such depictions were useful if a controller issued a traffic advisory relative to a ground feature such as 
an airport or VFR checkpoint (e.g., “Traffic . . . southeast-bound over Rockwall”). In the example shown 
the Ownship was not in a current or potential SS conflict with any intruders and no Stratway+ bands were 
present, but in cases where bands were present they would be redundantly displayed on the moving map’s 
airspeed tape, VVI and heading indicator rose as well as on the MACS PFD and ND, again for enhanced 
pilot situation awareness. 



Figure 5. CAS-1 Moving Map Display for UAS and Manned Aircraft Pilots 
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Figure 6. Close-Up View of Moving Map Display 

The aggregate objective of the PITL SS concept, primarily implemented by the Stratway+ bands but also 
with the converging-intruder alphanumeric display and moving map display as just described, is to restore 
“intelligent sight” (reference [2]) to the UAS pilot who does not have an out-the-window view of traffic 
and to enable that UAS pilot to make SS decisions and maneuvers in a manner comparable to the see-and- 
avoid decisions and maneuvers made by manned aircraft pilots. That is, during normal operations a 
manned aircraft pilot would execute see-and-avoid responsibilities by maintaining visual awareness of the 
traffic situation and exercising judgment based on that sight picture regarding any required ATC 
coordination and aircraft maneuvering to remain well clear of other aircraft. The PITL SS concept aims 
to enable a comparable level of normal-operations awareness and judgment for the UAS pilot, and in this 
sense the bands and other indications (e.g., amber intruder) are not necessarily intended as caution or 
warning alerts, but as SS guidance to enable normal-operations decisions and maneuvers. 

CAS-1 Experiment Description 


The CAS-1 experiment had both an exploratory and a quantitative objective. The exploratory objective 
was to evaluate the capability and viability of the prototype PITL SS concept instantiation, from the 
perspective of both subject air traffic controllers and CAS-1 staff research pilots, when the concept was 
exercised in a realistic, multi-aircraft simulation in light-to-moderate-density, moderate-complexity Class 
E airspace. The quantitative objective was to determine the range of HMD threshold values for the SSV 
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that was generally acceptable to air traffic controllers in this same Class E airspace. Significant effort was 
expended to generate simulation scenarios of sufficient complexity and traffic density to adequately 
represent today’s NAS environment and to effectively address these two objectives. That is, the research 
staff recognized that a given PITL SS concept and/or SSV HMD threshold might work acceptably in an 
isolated, pair-wise encounter between a UAS and an intruder but fall short when immersed in a more 
realistic environment, and wanted to take sufficient steps to test in that more realistic environment. 

CAS-1 was designed to address the following research questions for its quantitative objective, from the 
perspective of acceptability to air traffic controllers of a future environment with UAS DAA SS 
operations occurring in airspace shared with manned aircraft: 

1. What DAA SS horizontal maneuvers/HMD are too s mall, resulting in issuance of traffic safety 
alerts or controller perceptions of unsafe conditions? 

2. What DAA SS horizontal m aneuvers/HMD are too large (exc essive “well clear” dista nces), 
resulting in behavior the controller would not expect and/or disruptions to traffic flow? 

3. Is there a ran ge of acceptable DAA SS HMD that can be applied to the developm ent of DAA 
algorithms? Is this range affected by encounter geometry and/or speed differential, and if so, how? 

To address these research questions, 84 simulated SS encounters between a UAS and a manned aircraft 
were constructed with different encounter geometries, HMD, and speed differentials, and these SS 
encounters were then embedded throughout six one-hour simulated background traffic scenarios (14 
encounters per one-hour scenario) representative of light-to-moderate-workload TRACON traffic (both 
Instrument Flight Rules (IFR) and VFR traffic) on a calm-wind, clear-weather day. All of the SS 
encounters were constructed to occur in lower-altitude Class E airspace (most at 3000’ with some at 
4000’ or 6000’). This lower-altitude. Class E TRACON airspace is arguably some of the most 
challenging for a SS function, other than the traffic pattern area in the immediate vicinity of an airport, 
since it has significant complexity with arrivals, departures, over- flights, flight training, etc.; a mix of IFR 
and VFR traffic with some VFR aircraft not receiving air traffic services (i.e., not on the sector frequency 
and not subject to controller instructions); a high incidence of see-and-avoid/SS encounters; and 
significant traffic flow constraints that limit options for SS maneuvers, particularly for those requiring 
large HMD thresholds. 

All of the six one-hour scenarios (“Hours”) containing the SS encounters and background traffic were 
constructed in the DN/AR7 sector of the D10 TRACON. Dallas-Fort Worth International Airport (DFW) 
is the primary airport for the D10 TRACON; the DN/AR7 sector is in the northeast quadrant of D10 and 
handles south-flow traffic to/from satellite airports including Dallas Fove Field (DAF), Addison (ADS) 
and McKinney (TKI) as well as other non-towered airports and lower-altitude en route or training flights 
in the sector (see figure 7). Simulated UAS operations included arrivals to and departures from TKI as 
well as overflights throughout the sector, some with a SS encounter and some not. The physical locations 
of the encounters were varied, i.e., on departure, arrival, or en route in different parts of the sector, not 
only to avoid predictability of encounters in the later Hours but also to allow the six Hours to be 
constructed with the encounters embedded into realistic background traffic flows. For experiment control 
all UAS SS encounters were with VFR aircraft not receiving ATC services (i.e., not on the sector 
frequency) so a subject controller could not preemptively and strategically “fix” a SS encounter before it 
had a chance to occur. In most cases these encounters were also designed so that the controller could not 
see it developing far in advance; for example, the VFR intruder might be departing from a non-towered 
field and “pop up” on radar within a couple of minutes of CPA for the encounter, or alternatively might 
turn from a practice area toward the UAS shortly before CPA. 
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Figure 7. VFR Sectional Chart View - Approximate Location of DN/AR7 Sector 

The CAS-1 experiment data collection simulations were performed from January through March 2014 at 
NASA Langley Research Center’s Stinger Ghaffarian Technologies (SGT) contractor facilities at 130 
Research Drive in Hampton, Virginia. Fourteen retired air traffic control volunteers, all with DIO East- 
side training and experience, were recruited to control the simulated traffic scenarios and assess the 
acceptability of the different SS encounters. Most of the controllers had retired within the previous year 
and/or were still active as contract ATC instructors so were familiar with and proficient in DIO 
operations. None of the controllers had previous experience controlling UAS in their sectors. These ATC 
volunteers participated in CAS-1 data collection one-at-a-time, each for a two-day session. The first day 
of a session included approximately three hours of classroom and hands-on training and three one-hour 
data collection scenarios, and the second day included the remaining three one-hour scenarios and a 
debrief discussion of approximately one hour. 

During the one-hour scenarios all simulated UAS and manned aircraft receiving ATC services from the 
subject controller were “flown” by instrument-rated pilots as part of the simulation team, and all adjacent 
sector positions or local controller positions at the towered aiiports within DN/AR7 were staffed by 
retired air traffic controllers, also part of the simulation team. Additional traffic was also visible to the 
controller but not on the sector frequency, including VFR aircraft that were in the sector but not receiving 
services, landing traffic for which the controller had already approved a frequency change to the tower (or 
common traffic advisory frequency for non-towered airports), traffic in adjacent or overlying sectors, and 
traffic overflying the sector (e.g., DFW arrivals); all of this traffic was handled by simulation staffers 
and/or the MACS automation capabilities. 
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CAS-1 made the following assumptions for the simulation scenarios and the subject controllers were 
briefed accordingly during their training: 

1. A future UAS operating environment, consistent with assumptions in reference [5] and the FAA 
draft UAS C onOps, where appropriately certif ied DAA-equipped UAS w ould be operationally 
approved for operations under (only ) IFR with the DAA’ s PITL SS function replacing manned 
aircraft pilots’ see-and-avoid capability for the UAS while away from the immediate airport 
environment. Controllers were instructed to assume that the SS reliability was no better and no 
worse than current manned aircraft pilots’ see-and-avoid ability. 

2. Today’s ATC environment with no new UAS-specific operational improvements and only minimal 
changes to procedures and phraseology to accommodate UAS operations in the same airspace used 
by manned aircraft. 

3. Only nominal UAS DAA operations were simulated, i.e., no lost communications, lost C2 link or 
equipment failures were simulated and the communication and control latencies were minimized 
(approximately 100-150 ms communication delay which was the lower lim it of the simulation 
environment). 

CAS-1 independent variables included the encounter type (3 values: opposite-direction, overtake, or 
crossing), the SSV HMD (6 values: 0.5, 1.0, 1.5, 2.0, 2.5 or 3.0 nmi) and, for the crossing encounter type, 
the relative speed differential (5 values: intruder same speed as the UAS, or 40 or 80 knots slower or 
faster). This combination of independent variables resulted in 42 test conditions: 6 opposite-direction, 6 
overtake and 30 crossing encounter types. A replicate of each test condition was constructed, resulting in 
the 84 simulated encounters previously mentioned. 

The 42 test conditions were randomized in sequence and assigned 14 per hour to scenario Hours 1, 2 and 
3, which were presented to the subject controllers (in different orders, e.g., 123, 231, etc., for each 
subject) on Day 1, so for Day 1 Hours a subject might see one SS encounter with a 1.0 nmi HMD, the 
next with a 3.0 nmi HMD and the next with a 0.5 nmi HMD, etc. For Hours 4, 5 and 6 on Day 2 the 42 
replicate conditions were blocked with the same HMD values for a half-hour at a time, e.g., the first seven 
encounters in Hour 6 would all have 2.5 nmi HMD and the second seven encounters would all have 3.0 
nmi HMD, with the encounter type and speed differential randomized in sequence. This blocking of 
HMD for Hours 4-6 was done to see if workload ratings were affected if, for example, all UAS in the 
sector required a large HMD and thus potentially had a larger aggregate effect on traffic disruption. As 
with Hours 1-3, Hours 4-6 were presented on Day 2 in different orders to different subjects. 

The primary CAS-1 dependent metric was a direct assessment of HMD acceptability by the controller 
subjects immediately after each SS encounter. The assessment was based on a five-point scale as follows: 

1 . Much too close; unsafe or potentially so; cause or potential cause for issuance of a traffic alert 

2. Somewhat close, some cause for concern 

3. Neither unsafely close nor disruptively large, did not perceive the encounter to be an issue 

4. Somewhat wide, a bit un expected; might be disru ptive or p otentially disruptive in conge sted 
airspace and/or with high workload 

5. Excessively wide, unexpected; disruptive or potenti ally disruptive in congested airspace an d/or 
with high workload 

These assessments were elicited verbally and recorded by a retired controller subject matter expert (SME) 
who was part of the CAS-1 research team and sat next to each controller subject during the data collection 
runs. This SME was aware of each scripted SS encounter and was also monitoring the sector frequency 
and other subject controller tasks, so was able to elicit and record each assessment in a timely and non- 
obtrusive manner. Fractional assessments, e.g., “1.5” were allowed and sometimes assigned by subjects. 
Subjects were also encouraged, workload-permitting, to vocalize why they assigned a particular numeric 
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rating, particularly if the reason for the rating was non-obvious and/or if there were extenuating 
circumstances or additional remarks about the encounter. The controller SME also recorded any explicit 
action taken (vector, traffic advisory(s), safety alert, adjacent traffic re-route, etc.) by the subject related to 
each encounter, and noted any operational errors, deviations, significant voice communication errors, re- 
sequencing and/or other delays or unusual circumstances throughout all of the data runs. Additional 
dependent metrics included workload self-assessments using the Air Traffic Workload Input Technique 
(ATWIT) methodology at fixed intervals during the simulation and questionnaires at the end of each one- 
hour scenario. 

This paper reports results of the direct controller assessments of HMD, partial findings from controller 
comments during the debriefing sessions, and both subject controller and CAS-1 staff evaluations of 
concept viability. These partial CAS-1 results are presented in the next section. 


Partial Results and Discussion 


A total of 1 176 assessments of HMD were collected during the CAS-1 experiment across all subject 
controllers and data collection Hours. As expected there was some variability in controllers’ subjective 
assessments of acceptable HMD for SS encounters, particularly for small-HMD encounters, but there was 
a striking degree of agreement across all encounter geometries that a 1.5 nmi HMD was most acceptable 
(i.e., ratings at or near “3”: neither unsafely close nor disruptively large). More generally, most 
controllers found HMD values between 1 and 2 nmi acceptable across all encounter geometries. Figure 8 
shows average controller acceptability ratings for all opposite-direction (OD) encounters and these trends 
are apparent in this figure. 


Ratings by HMD (Opposite Direction) 



Horizontal Miss Distance (nmi) 

Figure 8. Controller Acceptability Rating Averages: Opposite Direction Encounters 


All OD encounters had a “geometric CPA” of zero if no SS action was taken, i.e. these were all initially 
collision course geometries requiring the UAS pilot to execute a SS maneuver in order to achieve the 
desired HMD with the respective intruder, and to deviate off course by the full amount of the respective 
HMD. Due to the high closure rates, a relatively large heading change was required for a SS maneuver to 
achieve a large HMD (e.g., 3 nmi), and this required heading change would quickly increase (i.e., heading 
bands would rapidly “grow”) if the pilot did not react soon after the first appearance of bands at the LAT 
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of 70 seconds. In some large-HMD cases there was not enough time for the UAS pilot to negotiate a SS 
maneuver with the controller before the bands indicated that a heading change of 40 degrees or more was 
required, and in these cases the UAS pilot made the maneuver first and informed the controller as soon as 
practicable afterward. Controllers stated in both their debrief comments and in contemporaneous 
comments immediately after large-HMD encounters that both the magnitude of deviation and the large 
heading changes were factors in their “excessively wide” ratings (i.e., ratings at or near “5”) for these 
encounters. They also noted that such large deviations and heading changes without an initial negotiation 
by the pilot contributed to the disruptiveness of the maneuver. Even in cases where the pilot negotiated 
the SS maneuver first, large-HMD maneuvers frequently required the controller to “point out” the traffic 
to adjacent sector controllers (e.g., when the UAS was near a sector boundary), question the UAS pilot 
about the required extent of his or her SS maneuver, issue traffic advisories to other aircraft in the vicinity 
or even to modify instructions to these aircraft (e.g., arresting a climb or descent through the UAS’s 
altitude or issuing a vector to keep separated from the UAS). 

The three-dimensional plot in Figure 9 shows more detail behind the average rating data shown in Figure 
8. This plot shows the frequency of a given rating for each of the six HMD values, and shows that nearly 
all of the 1.5 nmi HMD OD encounters received an acceptable “3” rating. The plot also shows the large 
number of “5” ratings (“excessively wide, unexpected; disruptive”) for 3 nmi HMD OD encounters. 

Ratings by HMD (Opposite Direction) 





Figure 9. Plot of Frequency of Rating Responses: Opposite Direction Encounters 


Much of the variability in controller subjective assessments of acceptable HMD occurred at the smaller 
end of the range, as can be seen on the plot in figure 9 for the 0.5 nmi HMD OD encounters. The plot 
shows that the average acceptability rating very near “2” (“somewhat close, some cause for concern”) 
shown in Figure 8 for 0.5 nmi HMD OD encounters is actually comprised of numerous “1” ratings 
(“much too close. . . unsafe. . . cause for issuance of a traffic alert”) as well as offsetting “3” ratings. This 
dichotomy was explored in some detail during the debrief sessions with the controllers and represents 
somewhat of a philosophical split in controller opinion. One cohort of controllers stated that if they had 
issued a traffic advisory to a (manned or UAS) pilot about an intruder and the pilot responded with 
“traffic in sight” (or “traffic detected” as the equivalent CAS-1 phrase used by SS-equipped UAS pilots), 
then the controllers felt that they had fulfilled their obligations and that subsequent separation from the 
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intruder was entirely the pilot’s responsibility (as supported by regulations), unless the pilot subsequently 
reported losing contact with the intruder. Another cohort stated that even if a pilot reported the traffic in 
sight (or detected for UAS) that the controllers still felt an obligation to monitor and intervene if the 
encounter appeared to be developing in an unsafe manner (one controller stated he felt a “moral 
obligation” to ensure safe operations in spite of the pilot’s legal obligation to ensure separation). This 
latter cohort tended to issue more “2” and “1” ratings to 0.5 nmi HMD encounters. Some of these 
controllers noted during the debrief sessions that they found these small-HMD encounters to be not only 
potentially unsafe but also disruptive, since they unnecessarily diverted controller attention away from 
other separation responsibilities elsewhere in his or her sector. It should be noted that nearly all of the 
participating controllers self-rated themselves during their debrief sessions as likely having a somewhat 
higher tolerance for small-HMD encounters than the controller population at large, due to their long 
experience with relatively high-density operations in the DIO TRACON. This subjective self-rating 
would tend to reinforce a conclusion that SS HMD values of 0.5 nmi for OD encounters may be too close 
for at least a subset of the controller population. 

Figures 10 and 1 1 show acceptability rating averages and frequency of rating responses, respectively, for 
overtake (OV) encounters. These were encounters where a UAS was overtaking a VFR intruder by 40 
knots on a collision (zero geometric CPA) course and deviated by the specified HMD value to stay clear 
of the intruder as it was passed. The results in these two figures show trends that are qualitatively similar 
to the OD encounters, but the small-HMD encounters have fewer “1” ratings and the large-HMD 
encounters have fewer “5” ratings. This phenomenon was explored to some extent in the debrief sessions 
and the most likely explanation is that the overtaking encounters had only a 40 knot closure rate and 
developed much more gradually than did the opposite-direction encounters. This likely led to a better 
comfort factor for controllers during a gradual, small-HMD overtake, and more time to compensate for 
the deviations of large-HMD overtakes. The slower closure rates also typically resulted in smaller, more 
acceptable heading changes to achieve the large-HMD deviations. 


Ratings by HMD (Overtake) 



Figure 10. Controller Acceptability Rating Averages: Overtake Encounters 


It should be noted from Figure 10 that there was slightly more rating variability from encounter to 
encounter with the same HMD, versus the OD encounter results shown in Figure 8. That is, the data 
points in Figures 8 and 10 each represent the average ratings across all 14 controllers for each encounter 
with a given HMD, even though those encounters occurred at different places in the sector across the six 
Hours. For example, as shown in Figure 8, the data points for all OD encounters with a 1.5 nmi HMD 
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basically lie on top of each other, with an average acceptability rating of “3” regardless of where the 
encounter occurred in the sector. In contrast, Figure 10 shows that some 1.5 nmi HMD OV encounters 
had average ratings slightly higher than “3” and others (occurring elsewhere in the sector) had ratings 
slightly lower than “3.” The reason for this slight variance is not immediately clear but is likely due to a 
deviation in one part of the sector being slightly more disruptive than in another part, due to traffic flow 
constraints and proximity to adjacent sector boundaries. Regardless of the reason, the OV encounter 
results in Figures 10 and 1 1 appear to support the conclusion that a l-to-2 nmi HMD range is most 
acceptable to controllers. 


Ratings by HMD (Overtake) 



Horizontal Miss Distance (nmi) 

Figure 11. Plot of Frequency of Rating Responses: Overtake Encounters 


Crossing (CR) encounters were treated somewhat differently than OD or OV encounters in two ways. 
First, the geometric CPA was non-zero for all CR encounters, so that if no SS action was taken the UAS 
would pass in front of the intruder by the amount of the geometric CPA. CR encounters with zero 
geometric CPA (i.e., collision course geometries) were not used because in almost all cases the obvious 
operational choice would then be to deviate behind the intruder, and technically there is no “well clear” 
requirement when passing behind an aircraft. However, there is a requirement to remain well clear when 
crossing in front of an aircraft, and setting up all CR encounters with non-zero geometric CPAs ensured 
that the SS encounters would result in the UAS passing in front of the intruder. The interactions between 
geometric CPA and SS HMD for CR encounters are explained further in the next paragraphs. 

The second way that CR encounters were treated differently than OD or OV encounters is that the speed 
differential between the UAS and intruder was manipulated as part of the construction of encounter 
geometries. That is, for each of the six SS HMD values, five CR encounters were constructed with the 
UAS either having the same speed as the intruder, or 40 or 80 knots slower or faster. In all CR 
encounters with a SS HMD of 1.5 nmi or less, the encounter was constructed with a geometric CPA equal 
to the HMD, so that no SS maneuver was required to achieve the HMD. For example, for all CR 
encounter cases with 0.5 nmi SS HMD the encounter geometry was set up with a 0.5 nmi geometric CPA, 
regardless of the speed differential between the UAS and intruder. In these cases the UAS would cross in 
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front of the intruder with 0.5 nmi HMD and with no SS maneuver action required or requested. If the 
subject controller issued a traffic advisory for the crossing traffic (almost always the case for small-HMD 
encounters) the UAS pilot would respond with either “traffic detected, no factor” or just “traffic detected” 
and take no action (unless subsequently issued a traffic alert instruction by the controller). 

Figures 12 and 13 show acceptability rating averages and frequency of rating responses, respectively, for 
CR encounters where the UAS was faster than its intruder. In all CR encounters where the UAS was the 
same speed or faster than its intruder and the HMD was greater than 1.5 nmi, the geometric CPA of the 
encounter was fixed at 1.5 nmi. For example, in the 3 nmi HMD CR encounter data shown in Figures 12 
and 13, the UAS would have crossed in front of the intruder by 1.5 nmi if no SS action was taken, but the 
bands commanded a further turn away from the intruder and the UAS pilot requested this turn to achieve 
a 3 nmi HMD (i.e., a 1.5 nmi deviation from course). 


Ratings by HMD (UA faster) 



The controller acceptability results shown in Figures 12 and 13 for CR encounter HMD are consistent 
with results for OD and OV encounters: HMD values between 1 and 2 nmi appear most acceptable to 
controllers. Figure 13 shows that even when the UAS was faster than its intruder, a significant number of 
“2” or “1” ratings resulted for 0.5 nmi HMD crossing encounters, indicating significant controller 
discomfort with these small HMD values. Figure 13 also shows a lower number of “5” ratings for 3 nmi 
HMD than resulted for OD or OV encounters, but this is likely due to the smaller total path deviations 
required to achieve 3 nmi of HMD in the CR encounters. That is, a 3 nmi HMD for an OD encounter 
would require a full 3 nmi deviation from the UAS flight path, whereas a 3 nmi HMD for a CR encounter 
would only require a 1.5 nmi deviation due to the 1.5 nmi geometric CPA built into the encounter 
geometry. 
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Ratings for Crossings (UA Faster) 



Horizontal Miss Distance (nmi) 


Figure 13. Plot of Frequency of Rating Responses: Crossing Encounters, UA Faster 

Figures 14 and 15 show acceptability rating averages and frequency of rating responses, respectively, for 
CR encounters where the UAS was slower than its intruder. In these cases the geometric CPA for the 
encounter was set equal to the SS FIMD for all values of HMD, so that no SS maneuver action was 
required to achieve FIMD in any case. There were two reasons for this decision. First, it was not clear a 
priori how much FIMD would be required for controller comfort when passing in front of a faster intruder 
(significantly faster in the 80 knot speed differential case), so the decision was made to explore FIMD 
values all the way out to 3 nmi to find this comfort-factor limit. Second, turning away from a much faster 
intruder to increase FIMD generally doesn’t work out well from an encounter geometry perspective and 
thus was not attempted when constructing scenarios. In real-world crossing encounters where insufficient 
distance would exist when passing in front of a faster intruder (e.g., a Piper Cub contemplating a cross in 
front of a Boeing 747 with insufficient FIMD) the maneuver is (hopefully) just not attempted, and the 
slower aircraft typically orbits until the faster aircraft passes, or passes (above and) behind. 
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Ratings by HMD (UA Slower) 



Figure 14. Controller Acceptability Rating Averages: Crossing Encounters, UA Slower 


The controller acceptability results shown in Figures 14 and 15 for small-HMD CR encounters are 
consistent with results for other encounters with small FIMD values. That is, a significant number of “2” 
or “1” ratings resulted for 0.5 nmi FIMD crossing encounters, indicating significant controller discomfort 
with these small FIMD values. Interestingly, most of the “too close” ratings disappeared for CR 
encounters with at least 1 to 1.5 nmi of HMD, even when passing in front of significantly faster intruders. 
Effectively none of the large-HMD encounters received a rating larger than “3” (i.e., no “too wide” 
ratings) because none of these encounters required a maneuver to achieve the large HMD. In other 
words, the UAS and intruder were just two aircraft in the airspace, flying past each other with a large 
HMD but no requested maneuver. The acceptability rating of “3” is appropriate in this case and fits the 
“3” rating description of “. . .did not perceive the encounter to be an issue.” The few “4” ratings for these 
large-HMD cases were due to misunderstanding of the training on this issue by the first subject, and 
subsequent training was modified to emphasize that encounters requiring no maneuver by the UAS 
should not receive ratings higher than “3” since by definition the non-maneuver is not disruptive. 
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Figure 15. Plot of Frequency of Rating Responses: Crossing Encounters, UA Faster 


As previously stated, the exploratory objective of CAS-1 was to evaluate the capability and viability of 
the prototype PITL SS concept instantiation, from the perspective of both subject air traffic controllers 
and CAS-1 staff research pilots. To that end, the subject controllers were asked during their debrief 
sessions if they thought the concept was viable from their perspective as a controller, with the caveat that 
the final SS HMD thresholds would be set to values that received acceptable (at or near “3”) ratings. The 
results were unanimously positive. Controller responses included the following: 

“definitely viable” 

“absolutely viable” 

“really impressed, way beyond expectations” [from before arriving and seeing the concept] 
[concept worked] “surprisingly well” 

“impressed with it” 

“don’t see any controller having an issue with” [the concept as presented in CAS-1] 

Many of the remarks such as these were volunteered by the controllers during their debrief sessions prior 
to being explicitly asked about concept viability. In hindsight these results are not overly surprising for 
several reasons. First, the PITL SS concept was intentionally designed to operate as similarly as possible 
to the see-and-avoid operations of manned aircraft pilots - from a subject controller’s perspective the only 
significant difference was phraseology, with “traffic detected” being used by UAS pilots instead of 
“traffic in sight” for manned aircraft. Second, controllers quickly realized that, as configured for CAS-1, 
UAS had perfect surveillance capabilities and always “detected” aircraft issued in a traffic advisory, so a 
sense of confidence was likely built that might not be as great in a real-world deployment of SS-equipped 
UAS that don’t always detect all intruders every time. Third, the UAS pilots were part of the CAS-1 staff 
and knew about all of the scripted encounters in advance, so any SS pilot blunders and “surprise factor” 
reactions that might occur in real-world operations were absent for CAS-1. Still, even with these caveats, 
the CAS-1 results provide an important validation of the PITL SS concept as outlined in reference [2] and 
instantiated as described in this paper. 
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CAS-1 staff pilots and researchers learned a great deal about operating details of the PITL SS concept in a 
real-world Class E airspace environment and these lessons are being applied to the spiral development of 
future concept refinements and instantiations. Two specific lessons are as follows: first, a means to 
reliably inform the pilot of time remaining, if any, to negotiate a SS maneuver with ATC would be highly 
advantageous. That is, Stratway+ bands appear when an intruder is within the LAT of penetrating the 
SSV, but a reliable indication of time remaining until the so-called SS Execution Threshold (SET), i.e., 
threshold beyond which no SS maneuver can avoid SSV penetration, did not exist in the CAS-1 
instantiation of the concept. Second, Stratway+ has no knowledge of intruder intent and this lack of 
knowledge can lead to some false alerts. A particularly troublesome example of these false alerts is the 
case of an aircraft nearby laterally that is descending to an altitude 500’ or 1000’ above the Ownship and 
then leveling. This will cause Stratway+ to provide bands across all 360 degrees of heading and bands on 
the VVI that indicate a descent is necessary; these bands all disappear (except for a “don’t climb” band on 
the VVI for an intruder 500’ above) once the intruding aircraft levels at its new altitude. In many of these 
cases the UAS pilot is aware of the intruder’s intent to level off, either by monitoring the party-line sector 
frequency and/or through explicit communication with the controller. Possible means for conveying this 
information to the Stratway+ SS guidance algorithm are under consideration for future refinements to the 
concept. 

Conclusions 


The data presented in the previous section show that when the UAS PITL SS capability described in this 
paper was simulated in moderate-complexity Class E airspace operations concurrent with light-to- 
moderate manned aircraft traffic, a SS HMD value of 1.5 nmi had the highest acceptability rating among 
all CAS-1 air traffic controller volunteers across all encounter geometries - opposite-direction, overtake, 
and crossing encounters with a wide range of speed differentials between the UAS and intruder aircraft. 

A total of 1 176 controller assessments were obtained across all encounter geometries with a striking 
degree of agreement among the controller volunteers about the acceptability of 1.5 nmi HMD. More 
generally, most controllers assessed HMD values between 1 and 2 nmi to be acceptable (neither unsafely 
close nor disruptively large) across all encounter geometries. There was more variability in controller 
assessments of 0.5 nmi HMD encounters but a significant number of these assessments considered this 
HMD value to be too small and potentially unsafe. HMD values larger than 2 nmi were generally 
assessed as increasingly disruptive to orderly traffic flow. These results should be useful to inform the 
development of operational performance standards for DAA SS functions. For example, it may be 
appropriate for standards to specify that SS functions should always indicate that SS maneuvering is 
necessary for encounters that will result in projected HMD values less than 1 nmi, and never indicate a 
required SS maneuver if the projected HMD is greater than 2 nmi. 

All 14 air traffic controller volunteers were favorably impressed with the PITL SS concept as simulated 
and presented to them, and considered the concept to be viable from an ATC perspective under the 
assumption that acceptable SS HMD values are employed. CAS-1 staff pilots also assessed the concept 
favorably and considered the simulation time valuable in identifying enhancements for future refinements 
of the concept. These controller and pilot assessments were significant since C AS-1 was the first 
sustained opportunity to exercise the Stratway+ instantiation of the PITL SS concept in a realistic 
controller-pilot simulation that was representative of the moderate-complexity, lower-altitude Class E 
airspace that will pose one of the most challenging environments for DAA-equipped UAS. 
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